For the original motivation, approach, and initial results, refer back to Experimental Optimizer Research & Progress. This update focuses on the progress made since that write-up.

Building on that earlier work, I continued refining the experimental optimizer while keeping its low-compute character.

The latest progress came from focusing on the geometry and conditioning of the matrix updates. This produced an improved experimental variant that reduces validation loss and closes more of the gap to Muon, without abandoning the original direction of the research.

Figure 1. Validation loss for Muon and the original and improved experimental optimizers.
Figure 1. Validation loss for Muon and the original and improved experimental optimizers.

Progress so far:

The improved Experimental (~7 GEMM) variant performs better than the original Experimental (~2 GEMM) variant and moves closer to Muon.

This improvement suggests that the remaining gap was influenced not only by the descent itself, but also by the geometry and conditioning of the updates. The research is still ongoing, and I am continuing to explore how much of this progress can be retained while keeping the optimizer efficient.

Additionally testing of the improved optimizer on post-training tasks is ongoing and proving fruitful and comparrable to Muon.